<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
 
 <title>papy - Google Code</title>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 
 <link rel="shortcut icon" href="http://www.gstatic.com/codesite/ph/images/defaultlogo.png"/>
 <link type="text/css" rel="stylesheet"
       href="./_static/custom.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_core.css" />
 <link type="text/css" rel="stylesheet"
       href="http://www.gstatic.com/codesite/ph/3770121226765569314/css/ph_detail.css" />
 <link rel="stylesheet"  type="text/css"
       href="./_static/pygments.css" /> 
 
 <link type="application/atom+xml" rel="alternate"
 href="/feeds/p/papy/svnchanges/basic?path=/wiki/" />
 
 
<!--[if IE]>
 <link type="text/css" rel="stylesheet"
       href="http://code.google.com/hosting/css/d_ie.css" />
<![endif]-->

<script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
          URL_ROOT:    '',
          VERSION:     '1.0b1',
          COLLAPSE_MODINDEX: false,
          FILE_SUFFIX: '.html'
      };
    </script>
    <script type="text/javascript" src="./_static/jquery.js"></script>
    <script type="text/javascript" src="./_static/interface.js"></script>
    <script type="text/javascript" src="./_static/doctools.js"></script>
    <link rel="contents" title="Global table of contents" href="contents.html" />
    <link rel="index" title="Global index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="top" title="papy v1.0b1 documentation" href="index.html" />
    <link rel="next" title="Installation" href="install.html" />
    <link rel="prev" title="PaPy - Parallel Pipelines in Python" href="index.html" />
</head>
<body class="t6">
 <div id=gaia>
  <font size="-1">
 
 <a href="http://code.google.com/p/support/wiki/WhatsNew" style="color:#a03">What's new?</a>
 | <a href="http://code.google.com/p/support/">Help</a>
 
 </font> 

 </div>
 <table style="padding:0px; margin: 0px 0px -6px 0; width:100%" cellpadding=0 cellspacing=0>
 <tr>
 <td><a href="/"><img src="http://www.gstatic.com/codesite/ph/images/defaultlogo.png" alt=papy/></a></td>
 <td>
 
 <div id="pname">
 <a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy</a>
 
 </div>
 <div id="psum">
 <i><a href="http://code.google.com/p/papy/" style="text-decoration:none; color:#000">papy v1.0b1 documentation</a></i>
 </div>
 
 </td>
 <td style="white-space:nowrap; text-align:right">
 
 <form action="http://code.google.com/hosting/search">
 <input size=30 name=q value=""/>
 <input type=submit name=projectsearch value="Search Projects" />
 <input type=submit name=websearch value="Search the Web" />
 </form>
 
 </tr>
 </table>


<table id="mt" cellspacing="0" cellpadding="0" width="100%" border="0">
 <tr>
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/">Project&nbsp;Home</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/downloads/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/downloads/list">Downloads</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/w/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/w/list">Wiki</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/issues/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/issues/list">Issues</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 
 
 
 <th onclick="if (!cancelBubble) _go('http://code.google.com/p/papy/source/list');">
 <div class="tab inactive">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <a onclick="cancelBubble=true;" href="http://code.google.com/p/papy/source/list">Source</a>
 </div>
 </div>
 </th><td>&nbsp;&nbsp;</td>
 
 
 
 <td width="100%">&nbsp;</td>
 </tr>
</table>
<table cellspacing="0" cellpadding="0" width="100%" align="center" border="0" class="st">
 <tr>
 
 
 
 <td>
 <div class="issueList">
<div class=isf>
 
 
 
 <a href="index.html">papy v1.0b1 documentation</a> |
 <span>
   <a href="genindex.html" title="General Index"
      accesskey="I">index</a>
 </span> |
 <span>
   <a href="modindex.html" title="Global Module Index"
      accesskey="M">modules</a>
 </span> |
 <span>
   <a href="install.html" title="Installation"
      accesskey="N">next</a>
 </span> |
 <span>
   <a href="index.html" title="PaPy - Parallel Pipelines in Python"
      accesskey="P">previous</a>
 </span> |
 <span class="inIssueList">
   <span>Search</span>
    <form action="search.html" method="GET" style="display:inline">
      <input size=30 name=q value=""/>
      <input type=submit name=projectsearch value="Search Documentation" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
 </span>

</div>
</div>

 </td>
 
 
 
 
 
 
 
 
 <td height=4 align=right valign=top class="bevel-right">
 <div class="round4"></div>
 <div class="round2"></div>
 <div class="round1"></div>
 </td>
 </tr>
</table>
<script>
 var cancelBubble = false;
 function _go(url) { document.location = url; }
</script>

<div id="maincol">
<!-- IE -->



<style>
 #downloadbox { 
 padding: 6px; 
 overflow: hidden;
 white-space: nowrap;
 text-overflow: ellipsis;
 }
 #downloadbox a {
 margin: 0 0 0 0;
 display: block;
 padding-left: 0px;
 }
 #owners a, #members a { white-space: nowrap; }
</style>

<div style="float:right; width:25em; margin:0 0 1em 2em">

 
 <div class=pmeta_bubble_bg>
 <div class="round4"></div>
 <div class="round2"></div>

 <div class="round1"></div>
 <div class="box-inner">
 <table class="pmeta" cellpadding=5>
 <tr><th>Previous:</th>
 <td><ul>
 
 <li>
   <a href="index.html" title="previous chapter"><strong>PaPy</strong> - Parallel Pipelines in Python</a>
 </li>
 

 
 </ul></td>
 </tr>
 <tr><th>Next:</th>
 <td><ul>
 
 <li>
   <a href="install.html" title="next chapter">Installation</a>
 </li>
 

 
 </ul></td>
 </tr>
 
 
 
 </table>

 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 
 
 <div style="background:#ddf8cc; margin-bottom: 5px;table-layout:fixed">

 <div class="round4"></div>

 <div class="round2"></div>
 <div class="round1"></div>
 <div class="box-inner">
 <div id=downloadbox>
  <b>Table of Contents</b><br/>
  <table class="pmeta toc" cellpadding=5><tr><td>
	<ul>
<li><a class="reference external" href="">Introduction</a><ul>
<li><a class="reference external" href="#where-when-should-papy-be-used">Where/When should <em>PaPy</em> be used?</a></li>
<li><a class="reference external" href="#where-when-should-papy-not-be-used">Where/When should <em>PaPy</em> <strong>not</strong> be used.</a></li>
<li><a class="reference external" href="#what-is-a-pipeline">What is a pipeline?</a></li>
<li><a class="reference external" href="#what-does-an-imap-do">What does an <em>IMap</em> do?</a></li>
<li><a class="reference external" href="#feature-summary">Feature summary:</a></li>
</ul>
</li>
</ul>

  </td></tr></table>

 
 </div>
 </div>
 <div class="round1"></div>
 <div class="round2"></div>
 <div class="round4"></div>
 </div>
 
 


 </div>
</div>

 <div style="padding:0 3em 1.2em 0">
   
     
  <div class="section" id="introduction">
<h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>
<p>Many computational tasks require sequential processing of data i.e. the global
task is split into sub-tasks which are evaluated in some order on the data to be
processed.</p>
<p><em>PaPy</em> allows to create a data-processing pipeline from components of isolated
functionality:</p>
<blockquote>
<ul class="simple">
<li>the function wrapping <em>Workers</em></li>
<li>the connection capable <em>Pipers</em></li>
<li>the topology defining <em>Dagger</em></li>
<li>the parallel executors <em>IMaps</em></li>
</ul>
</blockquote>
<p>The <em>Dagger</em> connects <em>Pipers</em> via pipes into a directed acyclic graph, while
the <em>IMaps</em> are assigned to <em>Pipers</em> and evaluate their <em>Workers</em> either locally
using threads or processes or on remote hosts. The <em>Workers</em> allow to compose
multiple functions while the <em>Pipers</em> allow to connect the inputs and outputs of
<em>Workers</em> as defined by the <em>Dagger</em> topology.</p>
<p>The input data can be processed in parallel if it is a collection (or it can be
split into one) of data-items: files, messages, sequences, arrays etc. (in
general all picklable python objects are valid input items).</p>
<p><em>PaPy</em> is written in and for Python this means that the user is expected to
write Python functions with defined call/return semantics, but the function code
is largely arbitrary e.g. they can call a perl script or import a library.
<em>PaPy</em> focuses on modularity, functions should be re-used and composed within
pipelines and <em>Workers</em>.</p>
<p>The <em>PaPy</em> pipeline automatically logs it&#8217;s execution is resistant to exceptions
and timeouts should work on all platforms where <tt class="docutils literal"><span class="pre">multiprocessing</span></tt> is available
and allows to utilize a cross-platform ad-hoc grid wherever <em>RPyC</em> is supported.</p>
<div class="section" id="where-when-should-papy-be-used">
<h2>Where/When should <em>PaPy</em> be used?<a class="headerlink" href="#where-when-should-papy-be-used" title="Permalink to this headline">¶</a></h2>
<p>It is likely that you will benefit from using <em>PaPy</em> if any of the following is
true:</p>
<blockquote>
<ul class="simple">
<li>you need to process large collections of data items.</li>
<li>your data collection is to large to fit into memory.</li>
<li>you want to utilize an ad-hoc grid.</li>
<li>you have to construct a complex data-flow.</li>
<li>you are likely to deal with timeouts or bogus data.</li>
<li>the execution of your workflow needs to be logged.</li>
<li>you want to refactor existing code.</li>
<li>you want to reuse(wrap) existing code.</li>
</ul>
</blockquote>
</div>
<div class="section" id="where-when-should-papy-not-be-used">
<h2>Where/When should <em>PaPy</em> <strong>not</strong> be used.<a class="headerlink" href="#where-when-should-papy-not-be-used" title="Permalink to this headline">¶</a></h2>
<blockquote>
<ul class="simple">
<li>The parallel features improve performance only if the functions have
sufficient granularity i.e. computation to communication ratio.</li>
<li>Your input is not a collection and it does not allow for data parallelism.</li>
</ul>
</blockquote>
</div>
<div class="section" id="what-is-a-pipeline">
<h2>What is a pipeline?<a class="headerlink" href="#what-is-a-pipeline" title="Permalink to this headline">¶</a></h2>
<p><em>PaPy</em> understands a pipeline as any directed acyclic graph. The direction of
the graph is defined by the data flow (edges, pipes) between the data-processing
units (nodes, <em>Pipers</em>).  To be precise the direction of the edges is opposite
to the direction of the data stream (pipes). Up-stream <em>Pipers</em> have incomming
edges from down-stream <em>Pipers</em> this is represented as a pipe with a opposite
orientation i.e. up-stream -&gt; down-stream. <em>PaPy</em> pipelines can be branched i.e.
two down-stream <em>Pipers</em> might consume input from the same up-stream <em>Piper</em> or
one down-stream <em>Piper</em> consumes data from several up-stream <em>Pipers</em>. <em>Pipers</em>
which consume (are connected to) data from outside the directed acyclic graph
are input <em>Pipers</em>, while <em>Pipers</em> to which no other <em>Pipers</em> are connected to are
output <em>Pipers</em>. <em>PaPy</em> supports pipelines with multiple inputs and outputs, also
several input nodes can consume the same external data. You should think of a
pipeline as an <tt class="docutils literal"><span class="pre">imap</span></tt> function composed <tt class="docutils literal"><span class="pre">imap</span></tt> functions i.e.:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># nested imaps as pipelines</span>
<span class="n">pipeline</span> <span class="o">=</span> <span class="n">imap</span><span class="p">(</span><span class="n">h</span><span class="p">,</span> <span class="n">izip</span><span class="p">([</span><span class="n">imap</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">input_for_f</span><span class="p">),</span> <span class="n">imap</span><span class="p">(</span><span class="n">g</span><span class="p">,</span> <span class="n">input_for_g</span><span class="p">)]))</span>
</pre></div>
</div>
<p>This is a pipeline of three functions f, g, h. Functions f and g are up-stream
relative to h. Because of the <tt class="docutils literal"><span class="pre">izip</span></tt> function input_for_f and input_for_g have
to be of the same lenght. In <em>PaPy</em> the lazy <cite>imap</cite> functions is replaced with
a pool implementation <em>IMap</em>, which allows for a parallelism vs. laziness
trade-off.</p>
</div>
<div class="section" id="what-does-an-imap-do">
<h2>What does an <em>IMap</em> do?<a class="headerlink" href="#what-does-an-imap-do" title="Permalink to this headline">¶</a></h2>
<p>The <tt class="docutils literal"><span class="pre">IMap.IMap</span></tt> object provides a method to evaluate a functions on a sequence
of changing arguments provided with optional positional and keyworded arguments
to modify the behaviour of the function. Just like <tt class="docutils literal"><span class="pre">multiprocessing.Pool.imap</span></tt>
or <tt class="docutils literal"><span class="pre">itertools.imap</span></tt> with the key differences that unlike <tt class="docutils literal"><span class="pre">itertools.IMap</span></tt> it
evaluates results in parallel. Compared to <tt class="docutils literal"><span class="pre">multiprocessing.Pool.imap</span></tt> it
supports multiple functions (called tasks), which are evaluated not one after
another, but in an alternating fashion. <em>IMap</em> is completely independent from
<em>PaPy</em> although they are boundled in a single Python package.</p>
</div>
<div class="section" id="feature-summary">
<h2>Feature summary:<a class="headerlink" href="#feature-summary" title="Permalink to this headline">¶</a></h2>
<p>This is a list of features of a pipeline constructed using the <em>PaPy</em> module components.</p>
<blockquote>
<ul class="simple">
<li>construction of arbitrarily complex pipelines</li>
<li>evaluation is lazy (buffered)</li>
<li>flexible local and remote parallelism</li>
<li>shared local and remote resources</li>
<li>robustness to exceptions</li>
<li>support for time-outs</li>
<li>real-time logging</li>
<li>os-independent (really a feature of <tt class="docutils literal"><span class="pre">multiprocessing</span></tt>)</li>
<li>cross-platform (really a feature of <em>RPyC</em>)</li>
<li>small code-base</li>
<li>a preliminary GUI based on <tt class="docutils literal"><span class="pre">Tkinter</span></tt></li>
<li>tested &amp; documented.</li>
</ul>
</blockquote>
</div>
</div>


   
 
 
 </div>
<div id="footer" dir="ltr">
 
 <div class="text">
 
 &copy;2008 Google -
 <a href="http://code.google.com/">Code Home</a> -
 <a href="http://code.google.com/tos.html">Terms of Service</a> -
 <a href="http://www.google.com/privacy.html">Privacy Policy</a> -
  
 </body>
</html>
